Multiple-Stage Knowledge Distillation

Knowledge distillation (KD) is a method in which teacher network guides the learning of student network, thereby resulting an improvement performance network. Recent research this area has concentrated on developing effective definitions knowledge and efficient methods transfer while ignoring ability To fully utilize potential improve efficiency, study proposes multiple-stage KD (MSKD) that allows students to learn delivered by multiple stages. The consists multi-exit architecture, imitate output at each exit. final classification achieved through ensemble learning. However, because results unreasonable gap between number parameters branch those as well mismatch capacity these two networks, we extend MSKD one-to-one method. experimental reveal proposed applied CIFAR100 Tiny ImageNet datasets exhibits good gain. enhancing changing style provides new insight into KD.